\section{Indexing}
\label{sec:index}
For indexing we have made a inverted index table \ref{sec:iv}, were we store each term together with the page's id and the frequency of the terms appearance. The page id points to a list which contains the pages crawled corresponding to the stored id. We chose the inverted index over a binary term matrix because it much more efficient regarding memory usage.

We receives a list of tokens from the \textit{Tokenizer}, a class we have constructed for generating the tokens, together with the crawled URL. These term tokens are then iterated through to check if they already exits in our index. If this is the case, the page is simply added to the corresponding \textit{term} together with the the frequency of the term appearance on the page. This can be seen on code snippet \ref{lst:indexer}.

\begin{code}{lst:indexer}{The \textit{addPage} method which takes care of storing the page correctly.}
\begin{lstlisting}
crawledPages.Add(url);
	foreach(string t in tokens.Distinct())
    {
    	if (termIndex.Where(x => x.termName == t).Count() == 0)
        {
        	termIndex.Add(new Term(t,crawledPages.IndexOf(url),tokens.Count(x => x.Equals(t))));
        }
        else
        {
        	int tmp = termIndex.FindIndex(x => x.termName.Equals(t));
            termIndex[tmp].addPageToTerm(crawledPages.IndexOf(url), tokens.Count(x => x.Equals(t)));
   	    }
	}
\end{lstlisting}
\end{code}

These values are stored in a list of \textit{KeyValuePairs} which we then can use to lookup the pages where the terms appears. This list together with the term's name are contained in a \textit{term} object. In the indexer we have a list containing every \textit{term} discovered through the crawling.

We created this structure because it gives us the possibility to easily see which page each term appeared on and how many times the term appeared on that specific page. The \textit{indexer} class also contains a list of crawled pages so we always know which web pages we already have crawled. In the \textit{indexer} we have a method that returns the number of pages a specific term have figured in, which we later use in the \textit{ranker} to calculate the score.